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Abstract — An  increasing  number  of  military  systems  are  being 
developed  using  service  orientation.  Some  of  the  features  that 
make  service  orientation  appealing,  like  loose  coupling,  dyna¬ 
mism  and  composition-oriented  system  construction,  make  secur¬ 
ing  service-based  systems  more  complicated.  We  have  been  de¬ 
veloping  technologies  for  Advanced  Protected  Services  (APS)  to 
improve  the  resilience  and  survival  of  services  under  cyber  at¬ 
tack.  These  technologies  introduce  a  layer  to  absorb,  contain,  and 
adapt  to  cyber  attacks  before  attacks  reach  critical  services.  This 
paper  describes  an  evaluation  of  these  advanced  protection  tech¬ 
nologies  using  cooperative  red  teaming.  In  cooperative  red  team¬ 
ing,  an  independent  red  team  launches  attacks  on  a  protected 
enclave  in  order  to  evaluate  the  efficacy  and  efficiency  of  the  pro¬ 
tection  technologies,  but  the  red  team  is  provided  full  knowledge 
of  the  system  under  test  and  its  protections,  and  is  given  escalat¬ 
ing  levels  of  access  to  the  system.  The  red  team  also  operates 
within  agreed  upon  rules  of  engagement  designed  to  focus  their 
effort  on  useful  evaluation  results.  Apart  from  presenting  the 
evaluation  results,  we  also  discuss  cooperative  red  teaming  as  an 
effective  means  of  evaluating  cyber  security. 
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I.  Introduction 

Service-orientation  is  a  software  engineering  methodology 
gaining  popularity  in  military  and  civilian  information  systems, 
many  of  which  play  important  roles  in  national  security.  Some 
of  the  very  features  that  make  service-orientation  appealing, 
like  loose  coupling,  dynamism  and  composition-oriented  sys¬ 
tem  construction,  make  securing  service-based  systems  more 
complicated.  These  features  ease  system  development,  but  in¬ 
troduce  additional  vulnerabilities  and  points  of  entry  beyond 
those  that  exist  in  self  contained,  static,  or  stove-piped  systems. 

We  have  been  developing  the  following  technologies  for 
Advanced  Protected  Services  (APS)  to  improve  the  resilience 
and  survival  of  services  under  cyber  attack: 

•  A  crumple  zone ,  analogous  to  the  crumple  zone  in  an  au¬ 
tomobile,  which  is  a  protective  layer  that  absorbs  the  ef¬ 
fects  of  attacks  by  localizing  or  eliminating  the  damage 
they  can  cause  and  leaving  the  critical  components  intact 
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and  unaffected. 

•  Containment  regions  which  ensure  that  attackers  have  to 
penetrate  layers  of  defenses  before  they  can  do  damage, 
and  that  multiple  flows  of  access  to  critical  services  are 
isolated,  so  that  successful  attacks  that  deny  service  to 
some  users  do  not  affect  others. 

•  Adaptive  response ,  so  that  attacks  whose  effects  might  be 
absorbed  by  the  crumple  zone  are  logged,  analyzed,  and 
blocked  on  subsequent  attempts.  This  helps  protect  against 
novel,  zero  day,  and  repeated  attacks. 

We  have  described  the  APS  crumple  zone  and  other  con¬ 
cepts  and  prototype  in  [1].  In  this  paper,  we  describe  the  exper¬ 
imental  evaluation  we  performed  with  the  US  Air  Force  Re¬ 
search  Laboratory  (AFRL)  to  assess  the  efficacy  of  the  APS 
protections.  The  evaluation  consisted  of  a  set  of  red  team  exer¬ 
cises ,  in  which  an  independent  set  of  penetration  testers  at¬ 
tacked  an  APS-protected  service  enclave  in  a  test  bed  at  AFRL. 
Provided  full  access  to  the  APS  source  code,  documentation, 
and  developers,  and  provided  with  ever-increasing  levels  of 
access  to  the  system  (including  insider  access),  the  red  team 
launched  a  variety  of  attacks  over  the  span  of  several  days.  The 
APS  system  held  up  against  these  attacks  well;  succumbing 
only  to  a  few  attacks  that  had  significant  insider  access  and 
privilege.  The  results  provide  insight  not  only  about  the  effec¬ 
tiveness  of  the  APS  survivability  approach  but  also  where  addi¬ 
tional  research  and  development  is  required. 

This  paper  provides  the  following  contributions: 

•  It  provides  important  information  about  the  effectiveness 
of  the  APS  approach  to  developing  survivable  systems. 

•  It  motivates  and  documents  the  use  of  cooperative  red 
teaming  as  an  important  part  of  security  evaluations.  This 
is  especially  important  because  system  security  is  notori¬ 
ously  difficult  to  measure  quantitatively  and  objectively. 

•  It  provides  a  development  and  evaluation  roadmap  for 
others  aspiring  to  develop  resilient  and  survivable  systems, 
and  further  research  directions. 

The  remainder  of  this  paper  is  structured  as  follows.  Sec¬ 
tion  II  provides  a  brief  introduction  to  the  APS  survivability 
prototype  system.  In  Section  III,  we  describe  the  cooperative 
red  teaming  approach.  In  Section  IV,  we  describe  the  specifics 
of  the  APS  red  team  exercise,  including  the  system  under  test, 
the  rules  of  engagement,  and  the  levels  of  access  granted  to  the 
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red  team.  In  Section  V,  we  summarize  the  results  of  the  red 
team  exercises.  In  Section  VI,  we  compare  to  related  research. 
Finally,  in  Section  VII,  we  present  some  concluding  remarks. 

II.  the  APS  Survivability  Architecture 

The  APS  protection  techniques  address  the  vulnerabilities  in¬ 
troduced  by  service-orientation  into  critical  systems.  While  the 
full  threat  model  that  we  target  is  beyond  the  scope  that  can  be 
covered  in  the  space  available  in  this  paper,  we  focus  on  the 
threats  enabled  by  the  open  discovery,  multiple  entry  points, 
and  dynamic  binding.  In  service-oriented  environments,  ser¬ 
vices  are  advertised  and  are  looked  up  by  potential  users,  many 
of  which  might  not  have  the  proper  authorization  to  access  or 
use  the  requested  services.  It  is  difficult  to  predict  at  design 
time  exactly  which  actors  will  attempt  to  consume  a  given  ser¬ 
vice  and  whether  they  will  be  authorized  to  do  so.  Network  and 
perimeter  security  only  reinforces  the  “crunchy  on  the  outside, 
chewy  inside”  view  of  software  systems,  and  are  utterly  insuf¬ 
ficient  for  developing  survivable  service-oriented  systems. 

We  introduce  a  new  architectural  feature,  a  crumple  zone 
(CZ)  that,  analogous  to  the  crumple  zone  in  an  automobile,  is 
positioned  in  front  of  critical  services  and  absorbs  the  effects  of 
attacks  by  localizing  or  eliminating  the  damage  they  can  cause 
and  leaving  the  critical  service  intact  and  unaffected. 

The  CZ,  shown  in  Figure  1  and  described  in  more  detail  in 
[1]  is  a  layer  of  intelligent  service  proxies  that  work  together  to 
present  a  high  barrier  of  entry  to  the  adversary,  to  increase  the 
chance  of  detection  of  malicious  activities,  and  to  contain  and 
recover  from  failures  and  undesired  conditions  caused  by  mali¬ 
cious  attacks.  These  proxies  collectively  implement  the  ser¬ 
vice’s  consumer-facing  interface.  Various  proxies  contain  ma¬ 
licious  activity  by  applying  security  checks  and  controls,  then 
approving  data  for  release  if  it  passes  those  checks.  Only  data 
that  has  been  inspected  and  approved  by  one  or  more  proxies  is 
passed  along  to  the  service.  Because  the  CZ  inspects  and  pro¬ 
cesses  untrusted  data,  it  is  expected  to  fail  occasionally.  There¬ 
fore,  we  include  monitoring  and  restart  of  CZ  components. 

To  be  effective,  all  client  interactions  with  the  protected 
service  must  be  intercepted  and  routed  through  the  CZ.  To 
make  the  CZ  non-bypassable,  we  use  several  techniques.  First, 
we  use  firewalls  and  routers  to  route  and  control  traffic  through 
the  CZ.  Second,  to  make  it  difficult  for  adversaries  to  discover 
and  access  protected  services,  CZ  presents  a  very  small  ex¬ 
ploitable  surface  to  untrusted  service  consumers  by  placing  the 


Figure  1 .  Architectural  Elements  of  the  Crumple  Zone 


crumple  zone  behind  a  firewall  that  uses  single  packet  authori¬ 
zation  (SPA)  [2].  Service  discovery  requests  that  make  it 
through  the  firewall  are  delivered  an  endpoint  reference  to  a 
termination  proxy  (TP)  that  is  used  as  the  entry  point  for  all 
incoming  client  connections. 

All  incoming  requests  are  escrowed  in  a  TP  that  terminates 
the  SSL  connection.  A  copy  of  the  request  is  forwarded  to  a  set 
of  proxies  that  implement  specific  checks  and  are  organized  in 
a  mechanism  proxy  cloud  (MPC).  We  have  implemented  prox¬ 
ies  that  check  assertions  on  application  data,  e.g.,  by  checking 
serialization  fields,  as  well  as  canary  proxies  that  consume  ap¬ 
plication  data  and  thereby  absorb  attacks,  e.g.,  by  crashing  or 
getting  corrupted. 

The  Splitter  component  replicates  SSL  streams  between  cli¬ 
ents  and  TPs  to  the  MPC  without  breaking  cryptographic  enve¬ 
lopes.  Key  management  components  of  the  CZ  selectively 
share  keys  from  the  TPs  to  the  MPC  so  that  the  new  streams 
can  be  decrypted  for  inspection. 

The  CZ  provides  both  horizontal  and  vertical  containment 
of  attacks.  Horizontal  containment  is  provided  by  the  layering 
of  multiple  and  varied  defenses  (i.e.,  providing  defense  in 
depth).  If  an  attack  gets  through  one  defense,  e.g.,  the  SPA 
firewall,  there  is  another  defense  there  to  block  it,  e.g.,  SELi- 
nux  policies  and  mechanism  proxy  checks.  Vertical  contain¬ 
ment  is  provided  by  multiple  MPCs,  each  running  in  their  own 
VM,  process,  or  dedicated  host.  Client  requests  are  spread  over 
these  MPCs  to  balance  and  contain  the  spread  of  attacks.  If  an 
attack  crashes  the  MPC  VM  or  process,  its  effects  will  be  local¬ 
ized  to  that  MPC  and  will  not  affect  client  interactions  routed 
through  other  MPCs. 

Lastly,  the  CZ  includes  adaptive  defense  for  attack  con¬ 
tainment.  CZ  components  store  audit  events  in  a  Log  Database. 
A  Log  Analysis  component  watches  for  violation  events,  which 
can  occur  when  a  mechanism  proxy  flags  a  suspected  attack, 
when  the  CZ  takes  an  attack  remediation  action,  or  when  an 
adverse  effect  of  an  attack  happens  (and  is  absorbed  by  the 
CZ).  For  violations  that  are  difficult  to  spoof,  the  Log  Analysis 
implements  a  simple  “three  strikes  and  you  are  out”  policy  by 
eventually  adding  source  IPs  of  connections  to  a  block  list. 

III.  The  Cooperative  Red  Team  Evaluation  Process 

Quantitative  evaluation  of  security  and  survivability  is  difficult. 
There  is  no  well-defined  and  generally  accepted  methodology. 
We  have  successfully  employed  red  team  experimentation  in 
previous  survivability  projects  [3] [4] [5] [6]  . 

A  Red  Team  exercise  is  a  controlled  experiment  to  assess 
security  and  survivability  of  a  defended  system  through  rigor¬ 
ous  testing,  performed  by  a  Red  Team  consisting  of  computer 
specialists  that  play  the  role  of  cyber  adversaries,  a  Blue  Team, 
consisting  of  system  operators  and  cyber  defenders,  and  a 
White  Team  that  plays  the  role  of  referee. 

Red  Team  exercises  can  be  structured  to  follow  cooperative 
or  adversarial  paradigms.  In  cooperative  arrangements,  both 
the  Red  and  Blue  Teams  work  together  to  focus  attacks  on  the 
parts  of  the  system  under  test  that  have  the  highest  chance  of 
yielding  useful  insights.  Also,  both  teams  work  together  to 
identify  the  right  mix  of  depth  vs.  breadth  of  attacks  to  maxim¬ 
ize  the  knowledge  that  can  be  learned  from  the  exercise,  which 
in  turn  feeds  into  the  next  technology  improvement  cycle.  An 
advantage  of  cooperative  exercises  is  that  the  developers  of  a 


system  can  participate  as  sophisticated  inside  attackers,  as  they 
already  know  everything  there  is  to  know  about  the  system, 
complementing  and  enhancing  the  Red  Team’s  knowledge  and 
expertise  with  the  latest  attacks  and  techniques.  In  adversarial 
exercises,  teams  do  not  cooperate  during  experiment  execution. 
Instead,  each  team  follows  their  objectives  to  either  defend  the 
system  under  test  or  attack  it.  While  adversarial  exercises  are 
considered  to  provide  more  realistic  and  independent  assess¬ 
ment,  previous  exercises  [7]  [8]  [9]  [10]  [11]  have  taught  us  that  a 
cooperative  methodology  is  more  valuable  and  cost-effective 
for  evaluating  R&D  prototypes. 

In  APS,  we  conducted  a  cooperative  red  team  exercise.  The 
Red  Team  consisted  of  members  of  an  independent  penetration 
testing  team,  other  AFRL  personnel,  and  “traitors”  from  the 
development  team.  The  Blue  Team  consisted  of  developers 
from  BBN  and  Adventium.  Finally,  the  Project  Managers  from 
AFRL  and  BBN  formed  the  White  Team.  Metrics  and  analysis 
from  the  red  team  exercise  complement  the  internal  laboratory- 
based  experimentation  conducted  under  the  APS  project. 

The  exercise  was  conducted  over  the  course  of  a  week  at 
AFRL,  after  several  months  of  cooperative  planning,  attack 
generation,  and  white  board  sessions.  During  the  preparation, 
BBN  provided  copies  of  the  software  with  documentation,  as¬ 
sisted  AFRL  in  setting  up  the  system  under  test,  and  provided 
in-depth  details  about  the  software’s  architecture  and  design. 

IV.  The  APS  Red  Team  Process 

A.  System  Under  Test(SuT)  and  Claims 

For  a  successful  assessment,  it  is  important  to  succinctly  de¬ 
scribe  the  system  to  be  evaluated  in  terms  of  the  technologies 
used,  component  layout,  and  network  topology  and  the  claims 
the  technology  is  making.  Figure  2  shows  the  components  that 
were  evaluated  as  part  of  the  exercise  together  with  Layer  4 
connections.  Figure  3  shows  the  network  topology  used  during 
the  experimentation.  To  assess  the  effectiveness  and  overhead 
of  the  technology  being  evaluated  the  SuT  needs  to  protect  a 
vulnerable  application,  which  serves  as  the  baseline  undefend¬ 
ed  system.  We  picked  a  baseline  undefended  application  based 
on  the  following  criteria.  First,  that  it  should  exhibit  vulnerabil¬ 
ities  that  are  commonly  found  in  DoD  systems,  e.g.,  threats 
listed  in  DoD  specific  threat  assessment  reports.  Second,  that  it 
is  susceptible  to  vulnerabilities  of  Information  Management 
Systems  (IMSs)  evaluated  in  previous  exercises  [8]  [9]  [10].  We 
compared  project  metrics  of  the  defended  system  (i.e.,  the  SuT 
configured  to  protect  the  vulnerable  application)  and  the  base¬ 
line  (i.e.,  a  configuration  containing  only  the  vulnerable  appli¬ 
cation),  to  validate  the  APS  survivability  claims. 

Claims  enable  us  to  focus  the  evaluation  on  specific  proper¬ 
ties  of  specific  parts  of  the  system  and  avoid  testing  the  space 
of  known  limitation.  We  defined  the  following  six  claims: 

1 .  The  CZ  improves  application  survivability,  including  con¬ 
fidentiality,  integrity,  and  availability  (CIA) 

2.  The  CZ  is  non-bypas sable,  meaning  all  traffic  between 
clientIP  and  jbossIP  is  intercepted  by  either  czIPl  or  czIP2 

3.  The  CZ  tolerates  crashes  of  a  single  instance  of  a  number 
of  components,  including  TPs,  Splitter,  RmiReg,  and  MPC 

4.  The  CZ  tolerates  corruption  of  the  Splitter  component 
aimed  at  impacting  message  integrity  and  confidentiality 


chentlp  czIPl  (lime)  czlP2  (grape)  jbossIP 

(aPS-laptop-02)  (loganberry) 


|^gy .  Processes  color  coded  by  implementation  language 

^  c  jvm  |  |  c Process  |  TCP  connection  SSL  Connection 

*:  Connections  to  LogDB  only  come  from  local  processes  and  are  omitted 
**:  Only  connections  to  one  NHC  are  shown,  but  they  exist  for  all  3  NHCs 


Figure  2.  Component  View 


5.  The  CZ  provides  controlled  inspection  of  application-level 
messages  while  maintaining  confidentiality  and  integrity 
of  data  (in  the  absence  of  corruption  of  CZ  components) 

6.  The  CZ  minimizes  information  disclosure  at  the  network 
layer  by  hiding  listening  sockets  to  unauthorized  clients. 

B.  Rules  of  Engagement  ( RoE ) 

Rules  of  engagement  are  used  to  guide  the  behavior  of  both  the 
Red  and  Blue  Teams.  We  used  RoE  to  avoid  testing  what  is 
already  known  about  the  SuT  and  to  focus  on  gaining  new  in¬ 
sights  instead.  For  cooperative  exercise,  it  is  less  important  to 
precisely  capture  the  RoE  than  it  is  to  document  known  limita¬ 
tions,  starting  points,  and  objectives  of  the  exercise. 

Known  Limitations:  The  following  list  contains  limita¬ 
tions  of  the  SuT  that  were  known  going  into  the  exercise  and 
hence  did  not  need  to  be  assessed: 


x 


Application  Server 


APS-controller 


Client-laptop-01  Client-laptop-02  Client-laptop-03  Client-laptop-06 


Figure  3.  Network  Topology  View 


Results  of  the  Red  Team  Exercises 


1.  No  redundant  network  paths.  Since  we  were  using  Single 
Small  Office  Home  Office  (SOHO)  routers,  we  ruled  out 
attacks  on  network  bandwidth  availability. 

2.  No  specific  router  settings.  Since  the  routers  were  not 
hardened,  e.g.,  by  configuring  static  ARP,  we  ruled  out 
low-level  network  attacks,  e.g.,  ARP  spoofing. 

3.  No  specific  OS  hardening  or  redundant  host  configuration. 
We  ruled  out  attacks  targeted  at  corrupting  or  crashing  the 
CZ  TCP/IP  stack. 

4.  No  defenses  for  the  client  machine.  We  ruled  out  attacks 
against  the  client.  In  fact,  a  corrupt  client  was  an  attacker 
starting  point. 

These  limitations  were  due  to  the  fact  that  we  were  evaluating  a 
R&D  prototype  system  rather  than  a  transition-ready,  matured 
and  hardened  system.  None  of  the  limitations  listed  here  re¬ 
quire  new  R&D  efforts,  and  can  be  addressed  by  simply  taking 
existing  technologies  and  spending  the  time,  effort,  and  budget 
to  harden  and  configure  them  appropriately. 

Adversary  Starting  Points:  To  perform  a  thorough  as¬ 
sessment  of  a  survivable  system,  it  is  important  to  avoid  situa¬ 
tions  where  the  Red  Team  spends  a  large  amount  of  time  trying 
to  overcome  and  evaluate  only  a  specific  defense  as  opposed  to 
an  overall  evaluation  of  the  system,  especially  for  a  system  that 
is  designed  with  defense-in-depth  in  mind.  For  instance,  in  the 
OASIS  Dem/Val  Red  Team  exercise  [4],  the  Red  Teams  spent 
days  working  against  network-level  protections.  Since  the  at¬ 
tacker  laptop  hooked  up  to  the  WAN  (analogous  to  the  starting 
point  1  below)  was  their  only  starting  point,  this  necessary  first 
step  took  time  away  from  other  useful  testing. 

For  the  purpose  of  the  APS  exercise,  the  Red  Team 
launched  attacks  from  the  following  starting  points  with  vary¬ 
ing  privileges,  enabling  us  to  assess  the  multiple  levels  of  de¬ 
fense  provided  by  APS: 

1.  Attacker  Machine:  The  adversary’s  machine  (i.e.,  the  ad¬ 
versary  has  root  access)  is  connected  to  one  of  the  client 
enclaves. 

2.  Corrupted  Client:  The  adversary  has  root  access  on  a  client 
machine. 

3.  Corrupted  TP:  The  adversary  can  execute  arbitrary  code  as 
part  of  the  TP  process,  realized  by  allowing  the  adversary 
to  add  custom  classes  to  the  TP’s  classpath.  Note  that  this 
does  not  equate  to  root  privilege  on  the  TP  machine. 

4.  Corrupted  MPC  process:  The  adversary  can  execute  arbi¬ 
trary  code  as  part  of  a  constituent  MPC  process. 

Threat  Model:  The  threat  model  was  based  on  the  notion 
of  sophisticated  adversaries.  We  assume  that  the  adversary  has 
complete  knowledge  about  the  system,  including  access  to 
network  topology  diagrams,  architecture  and  design  docu¬ 
ments,  and  all  source  code.  However,  the  adversary  does  not 
have  access  to  locally  installed  crypto  material  on  end-systems, 
e.g.,  private  keys  used  to  establish  TFS  connections  or  pass 
SPA  authorization.  The  overall  objective  of  the  Red  Team  is  to 
assess  the  effectiveness  of  the  CZ  by  trying  to  cause  loss  of 
availability,  integrity,  or  confidentiality  of  the  protected  ser¬ 
vice.  The  objective  of  the  Blue  Team  is  to  keep  up  service 
availability,  integrity,  and  confidentiality  in  the  presence  of 
sustained  attacks  by  participating  in  the  operations  of  the  CZ 
via  administrative  interfaces. 


V. 

Over  the  course  of  one  week  at  the  Air  Force  Research  Fabora- 
tory  in  Rome,  NY,  the  red  team  launched  over  40  attacks  on 
the  APS  protected  service.  While  the  attacks  that  were 
launched  represent  a  specific  attack  scenario,  the  attacks  pro¬ 
vided  coverage  over  commonly  used  threat  models  for  web 
applications,  e.g.,  the  OWASP  Top  10  list.  In  many  cases, 
when  an  attack  did  not  succeed,  we  provided  more  access  (e.g., 
turning  off  a  firewall  or  launching  the  attack  from  inside  the 
CZ  host  and  tried  the  attack  again).  In  addition,  we  conducted 
white  board  sessions  for  a  dozen  additional  attacks,  i.e.,  we 
discussed  what  the  effects  would  be  and  analyzed  the  effec¬ 
tiveness  of  the  attack,  the  defense  against  it,  and  decided 
whether  the  attack  should  be  launched. 

The  test  environment  included  automated  client-bots  mak¬ 
ing  continuous  calls  on  the  protected  service  through  the  CZ. 
During  the  attack,  blue  team  members  looked  over  system  logs 
in  real  time,  but  did  not  take  any  defensive  action. 

The  APS  protected  system  exhibited  significant  resilience 
and  adaptability  at  multiple  layers.  No  attack  without  privilege 
or  only  client  privilege  succeeded.  Two  attacks  succeeded  in 
causing  damage  as  planned  without  additional  privilege,  one 
that  corrupted  the  splitter  and  one  that  launched  a  fork  bomb , 
i.e.,  an  attack  that  forks  new  processes  recursively  until  all 
CPU  resources  are  exhausted.  Both  attacks  started  inside  and 
with  considerable  privilege.  Three  other  attacks  eventually  suc¬ 
ceeded  when  granted  deeper  access  and  additional  privilege. 

Table  1  shows  a  summary  of  the  attacks  executed  during 
the  weeklong  red  team  exercise.  It  shows  the  attack  number, 
variants  if  any,  a  description  of  the  attack,  which  of  CIA  it  co¬ 
vers,  which  APS  claims  it  covers  (described  in  Section  IV),  and 
whether  the  attack  runs  were  successfully  survived  or  not.  The 
attacks  were  determined  by  members  of  both  red  and  blue  team 
based  on  the  threat  model  and  prior  team-member  experiences 
aiming  to  achieve  coverage  over  the  APS  claims. 

As  Table  1  shows,  we  ran  multiple  runs  for  some  attacks,  in 
most  cases  granting  increasing  access  and  privilege.  In  others, 
the  additional  runs  were  testing  different  starting  points.  A 
green  check  mark  means  the  attack  did  not  succeed,  i.e.,  the 
APS  software  absorbed  the  effects  of  the  attack.  The  yellow 
triangle  (a  yield  sign)  means  that  the  attack  worked,  but  only 
when  additional  concessions  (such  as  greater  access  or  shutting 
off  some  defenses)  were  provided  to  the  attackers.  A  red  X 
means  that  the  attack  worked  as  planned. 

Despite  the  successful  defense  against  the  attacks  that  had 
considerable  access  and  privilege,  we  note  that  highly  effective 
attacks  that  we  did  not  cover  or  consider  may  very  well  exist. 
But  overall,  we  have  taken  a  significant  step  in  the  right  direc¬ 
tion  improving  the  defense  of  service-oriented  systems. 

Time  and  resources  did  not  permit  running  all  attacks  that 
were  conceived  and  developed  to  varying  degrees  of  maturity. 
Some  of  them  were  run  outside  the  context  of  the  red  team  ex¬ 
ercise  week  at  the  blue  team  laboratory.  Others  were  analyzed 
during  white  board  sessions  at  various  planning  meetings  lead¬ 
ing  up  to  and  during  the  exercise  week.  Table  2  shows  the 
summary  of  runs  that  were  performed  outside  of  the  exercise 
week  and  attacks  that  were  analyzed  during  white  board  ses¬ 
sions.  The  legends  are  same  as  Table  1,  except  the  “NA”,  indi¬ 
cating  that  the  attack  was  not  applicable  to  the  service  used  in 
the  system  under  test  (e.g.,  the  attack  exploited  a  vulnerability 


Table  1.  Summary  Results  of  Red  Team  Experiments  Ran  during  the  Exercise  Week. 


No. 

Var 

Description 

CIA 

Claim 

Survival  Against  Runs 

4 

SQU-  quote-truncation 

C 

5 

T 

10 

Watchdog- kill  watched  process  (a-c) 

A 

3 

X 

14 

NW  Recon- sniff  identify  connection  and  data 

C 

1 

15 

Corrupt  splitter  performing  malicious  acts(a-g) 

C,I,A 

4 

X 

X 

16 

App  Rate-maliciousclientmakingtoo  manyrq 

A 

1 

17 

App  Size-trying  to  allocate  large  memory 

A 

1 

18 

FA  Bomber-tryingto  access  files  from  MP 

C 

1,5 

19 

SQL  123-  multiple flavorsof SQL  injection 

C 

5 

20 

Run  without  SPA  (controlfor  degraded  config.) 

A 

1,6 

21 

Serialization- system  exit  while  deserialize 

A 

3 

22 

Whitelist- sneak  in  unauthorized  object 

A,l 

1,3 

23 

Log  Analysis- 3  strikes  to  adaptive  response 

C,I,A 

1 

24 

1 

TCP  Flood  no  knock:  too  many  req  w/  o  knock 

A 

1 

2 

TCP  Flood  w  knock  LDAP  and  RMI  ports(a-b) 

A 

1 

3 

TCP  Flood  LDAP  and  RMI  after  SPA  (a-b) 

A 

1 

25 

Direct  Connection  from  ClienttoJBoss 

l,C 

2 

26 

1 

Dir  Con  toJBoss  from  MP  w/wo  JVM  Pol.(a-d) 

l,C 

2 

X 

2 

Same  as  above,  &  try  to  exec  code  NH  (a-e) 

l,C 

2 

X 

27 

Fork  Bomb  (a-b) 

A 

1,3 

X 

▼ 

28 

Multi  user: stress  CZs  with  client  load  (a-g) 

A 

NA 

▼ 

▼ 

▼ 

▼ 

x 

NA 


Table  2.  Summary  Outcome  of  Other  Attacks. 


No. 

Var 

Description 

CIA 

Claim 

2 

SQLI-a:  Insert  records  into  the  services  db 

1 

5 

3 

l 

SQLI-b:  see  if  db  content  with  apostrophe  isdenied 

A 

5 

NA 

2 

SQLI-b:  deny  client  by  db  violation  with  spoofed  credential 

A 

5 

NA 

6 

LogDB-c:  corrupt  the  log  dbto  force  a  bad  action/cover  track 

1 

5 

7 

SPA-self-auth: 

C,I,A 

5 

8 

Corrupt  LDAP 

C,I,A 

1 

13 

1 

Use  backtrack  and  metasploitto  escalate  access,  remote  shell 

C,l 

1 

2 

Same  as  above  +  the  vulnerable  class  is  white  listed 

C,l 

1 

3 

Same  as  above  +  JVM  policies  weakened  to  allow  remote  con 

C,l 

1 

4 

Same  as  above  +  JVM  policies  turnedoff 

C,l 

1 

5 

Sameas  above+  SEL  weakened  to  allow remotecon 

C,l 

1 

▼ 

6 

Sameas  above+  SELTurned  off 

C,l 

1 

X 

S  Attack  success¬ 
fully  absorbed 

▼  Attack  succeed¬ 
ed  after  conces¬ 
sions  granted  to 
attackers 

X  Attack  not  sur¬ 
vived;  i.e.,  the  at¬ 
tack  succeeded 


Attack  successfully  ab¬ 
sorbed 

Attack  succeeded,  but  only 
after  concessions  granted 
to  attackers 

Attack  not  survived;  i.e., 
the  attack  succeeded 

Attack  not  applicable  to  the 
protected  service 


that  did  not  exist  in  the  service  being  protected). 

In  addition,  other  attacks  were  not  run  for  various  reasons. 
Attack  1  was  an  example  used  to  illustrate  the  red  team  process 
and  is  not  included  in  any  results.  Attack  5  was  deemed  to  be 
impossible  to  launch.  Finally,  attacks  9,  11,  and  12  were 
deemed  to  be  duplicates  of  other  attacks  that  were  run. 

Combining  the  actual  runs  and  white  boarded  attacks  pro¬ 
vide  a  good  test  and  evaluation  coverage  over  the  CIA  triad  and 
the  APS  survivability  claims.  Many  attacks  were  generic,  i.e., 
depending  on  the  payload  they  could  attempt  to  compromise  a 
combination  of  C,  I  or  A  of  some  aspect  of  the  protected  ser¬ 
vice  or  the  APS  defense — these  runs  were  counted  in  all  three 
(C,  I,  and  A)  categories.  The  total  number  of  runs  attempted  to 
compromise  each  of  the  C,  I,  A  attributes  of  some  aspect  of  the 
system  and  how  they  fared  are  shown  in  the  left-hand  side  chart 


of  Figure  4.  The  right  hand  chart  shows  unique  attacks  and 
attack  runs  aimed  at  the  specific  individual  APS  survivability 
claims  and  how  they  fared.  ”F”  and  the  green  color  indicate  the 
number  of  attacks  that  failed,  i.e.,  did  not  work  as  planned. 
“P/S”  and  the  yellow  color  indicates  the  number  of  attacks  that 
were  partially  successful,  i.e.,  that  worked  after  additional  con¬ 
cession  was  provided  to  the  attackers.  “S”  and  the  red  color 
indicate  the  number  of  attacks  that  succeeded. 

VI.  Related  Work 

Red  Teams,  defined  as  “an  organizational  element  comprised 
of  trained  and  educated  members  that  provide  an  independent 
capability  to  fully  explore  alternatives  in  plans  and  operations 
in  the  context  of  the  operational  environment  and  from  the  per- 
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■ 

F 

P/S 

s 

c 

18 

4 

1 

18 

3 

5 

A 

22 

6 

Claim 

1 

2 

3 

4 

5 

6 

Attacks 

Total 

12 

3 

4 

1 

6 

1 

Runs 

Total  20 

10  7 

6 

6  1 

F/  /S 

15/  /  2 

8/  /  2 

4/  /  2 

3/  /  2 

4/  /0 

1/  /O 

*  Actual  and  whiteboarded 

Figure  5.  Map  of  Test  and  Evaluation  Coverage  Based  on  the  Attacks  Ran  and  Analyzed  during  White  Board  Sessions. 


spective  of  adversaries  and  others”  [12],  are  not  a  new  concept 
in  cyber  security.  Various  commercial  and  DoD  organizations 
as  well  as  national  labs  offer  Red  Teaming  service  and  training. 
The  Sandia  Information  Design  Assurance  Red  Team  (ID ART) 
[13]  is  fairly  well  known  and  is  the  process  that  the  AFRL  pen¬ 
etration  testing  team  followed  during  the  APS  exercises.  The 
U.S.  Army  University  of  Foreign  Military  and  Cultural  Studies 
(UFMCS)  [14]  also  offer  a  course  on  Red  Teaming. 

Penetration  testing  is  related  to  Red  Teaming,  although  the 
exact  definition  and  distinction  from  Red  Teaming  is  murky. 
We  see  penetration  testing  as  “security  testing  in  which  evalua¬ 
tors  attempt  to  circumvent  the  security  features  of  a  system 
based  on  their  understanding  of  the  system  design  and  imple¬ 
mentation”  [15].  As  such,  it  is  expected  that  a  Red  Team  Exer¬ 
cise  will  include  some  level  of  penetration  testing.  The  Open 
Web  Application  Security  Project  (OWASP)  Testing  Guide 
[16]  illustrates  a  number  of  penetration  testing  techniques  that 
are  commonly  used  against  web  application  to  assess  security. 

VII.  Conclusions 

We  described  a  successful  application  of  cooperative  red  team¬ 
ing  to  evaluate  the  R&D  prototype  of  a  survivable  services 
architecture.  We  argue  that  cooperative  red  teaming  is  more 
effective  for  research  prototypes  than  other  red  team  approach¬ 
es,  such  as  an  adversarial  red  team.  The  cooperative  approach 
supports  more  cost-effective  evaluation  by  sharing  information 
that  focuses  on  attacks  that  were  likely  to  have  an  adverse  ef¬ 
fect  on  the  system  and  discard  attacks  (after  analysis  and  dis¬ 
cussion)  that  would  have  little  or  no  effect.  The  red  team  was 
able  to  quickly  gain  a  deep  understanding  of  the  system  under 
test  due  to  full  access  to  the  software  and  developers.  The  red 
team  had  full  knowledge;  there  was  no  attempt  to  gain  security 
by  obscurity.  The  red  team  did  not  have  to  waste  time  on  pene¬ 
tration  attacks  to  gain  access  to  launch  sophisticated  attacks  - 
privileged  and  insider  starting  points  were  granted  to  them. 

Likewise,  the  combination  of  outsider  view  and  insider  in¬ 
sight  in  the  cooperative  red-team  setting  led  to  better  learning 
and  discovery  of  nooks  and  crannies  of  the  APS  architecture 
and  its  defenses.  One  of  the  significant  results  from  this  exer¬ 
cise  was  how  well  the  APS  software  held  up  against  the  sus¬ 
tained  and  new  attacks  the  red  team  launched.  This  validates 
the  APS  approach  and  motivates  further  research,  development, 
and  evaluation  of  the  crumple  zone  architecture  for  survivabil¬ 
ity.  In  addition,  successful  attacks  highlighted  places  where 
additional  research,  development,  and  evaluation  are  needed. 

As  the  APS  prototype  technology  matures  and  becomes 
more  transition-ready,  we  expect  additional  experimental  eval¬ 
uation  of  the  technology  deployed  in  target  environments. 
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